Evaluation of Coordination Techniques in Synchronous Collaborative Information Retrieval

Traditional Information Retrieval (IR) research has focussed on a single user interaction modality, where a user searches to satisfy an information need. Recent advances in web technologies and computer hardware have enabled multiple users to collaborate on many computer-supported tasks, therefore there is an increasing opportunity to support two or more users searching together at the same time in order to satisfy a shared information need, which we refer to as Synchronous Collaborative Information Retrieval (SCIR). SCIR systems represent a significant paradigmatic shift from traditional IR systems. In order to support effective SCIR, new techniques are required to coordinate users’ activities. In addition, the novel domain of SCIR presents challenges for effective evaluations of these systems. In this paper we will propose an effective and re-usable evaluation methodology based on simulating users searching together. We will outline how we have used this evaluation in empirical studies of the effects of different division of labour and sharing of knowledge techniques for SCIR.